Methods, systems, and media for storing information associated with content presented on a media presentation device

ABSTRACT

Methods, systems, and media for performing personalized actions on mobile devices associated with a media presentation device are provided. In some implementations, the method comprises: determining that a mobile device is associated with a media presentation device, wherein the media presentation device is in an operating mode that presents a sequence of images; receiving a user command that indicates interest in an image from the sequence of images; determining metadata associated with the image; causing the metadata to be stored with a representation of the image and a user account identifier; receiving a request to review saved image information corresponding to one or more images in which the user command indicative of interest has been received; and, causing a user interface element to be presented that allows a user of the mobile device to navigate through the saved image information.

TECHNICAL FIELD

The disclosed subject matter relates to methods, systems, and media forstoring information associated with content presented on a mediapresentation device.

BACKGROUND

While consuming media content being displayed on a television device, aviewer of the media content is often interested in information relatingto the media content, such as additional information about a locationrelated to the media content, information about a topic related to themedia content, etc. For example, the viewer may want to search foradditional information about an actor seen in a movie or about locationsthat are shown in a documentary.

Search engines allow the viewer to search on the Internet for resources,which include webpages, images, video content, and audio content. Atypical search engine provides a web page for entering search terms,where the search terms are inputted by the viewer. Search resultsconducted for particular search terms are provided to the viewer.Moreover, with the advent of mobile devices, the use of mobile searchengines has increased. For example, using a mobile device, the viewermay perform a search and browse through Internet content while watchingthe television program.

Text-based search engines, however, may be somewhat limited for thisuser. In an example where the user wants to learn the name of aparticular actor or actress in a movie and obtain additional informationabout the particular actor or actress, the user may attempt to obtainthis information by finding the name of the movie (e.g., from a programguide, from an online television guidance website, from a printed guide,etc.) and searching for the movie by inputting it into a mobile searchengine. The user then clicks through various search results to find awebpage containing information relating to the movie (e.g., an onlinewebsite containing information relating to movies and televisionprograms), accesses the webpage, and searches through the webpage tofind information relating to the particular actor or actress. If theuser is able to find the actor or actress on the webpage, the useraccesses another page on the website relating to the particular actor oractress and scans through this page to find the desired informationabout the particular actor or actress. This is a time consumingprocedure for the user. Moreover, this may cause the user to miss asubstantial portion of the television program or to pause the televisionprogram to obtain such information via this procedure.

Accordingly, new methods, systems, and media for storing informationassociated with content presented on a media presentation device aredesirable.

SUMMARY

In accordance with some implementations of the disclosed subject matter,mechanisms for storing information associated with content presented ona media presentation device are provided.

In accordance with some implementations of the disclosed subject matter,a method for obtaining information relating to presented content isprovided, the method comprising: determining that a mobile device isassociated with a media presentation device over a communicationsnetwork, wherein the media presentation device is in an operating modethat presents a sequence of images; receiving a user command thatindicates interest in an image from the sequence of images beingpresented on the media presentation device; determining metadataassociated with the image from the sequence of images being presented onthe media presentation device; causing the metadata associated with theimage to be stored with a representation of the image from the sequenceof images and a user account identifier corresponding to a user of themobile device; receiving a request to review saved image informationcorresponding to one or more images in which the user command indicativeof interest has been received; and in response to receiving the request,causing a user interface element to be presented that allows a user ofthe mobile device to navigate through the saved image information,wherein the user interface element includes at least the representationof the image from the sequence of images and information relating to theimage from the sequence of images.

In accordance with some implementations of the disclosed subject matter,a system for obtaining information relating to presented content isprovided, the system comprising a hardware processor that is configuredto: determine that a mobile device is associated with a mediapresentation device over a communications network, wherein the mediapresentation device is in an operating mode that presents a sequence ofimages; receive a user command that indicates interest in an image fromthe sequence of images being presented on the media presentation device;determine metadata associated with the image from the sequence of imagesbeing presented on the media presentation device; cause the metadataassociated with the image to be stored with a representation of theimage from the sequence of images and a user account identifiercorresponding to a user of the mobile device; receive a request toreview saved image information corresponding to one or more images inwhich the user command indicative of interest has been received; and inresponse to receiving the request, cause a user interface element to bepresented that allows a user of the mobile device to navigate throughthe saved image information, wherein the user interface element includesat least the representation of the image from the sequence of images andinformation relating to the image from the sequence of images.

In accordance with some implementations of the disclosed subject matter,a non-transitory computer-readable medium containing computer executableinstructions that, when executed by a processor, cause the processor toperform a method for obtaining information relating to presented contentis provided, the method comprising: determining that a mobile device isassociated with a media presentation device over a communicationsnetwork, wherein the media presentation device is in an operating modethat presents a sequence of images; receiving a user command thatindicates interest in an image from the sequence of images beingpresented on the media presentation device; determining metadataassociated with the image from the sequence of images being presented onthe media presentation device; causing the metadata associated with theimage to be stored with a representation of the image from the sequenceof images and a user account identifier corresponding to a user of themobile device; receiving a request to review saved image informationcorresponding to one or more images in which the user command indicativeof interest has been received; and in response to receiving the request,causing a user interface element to be presented that allows a user ofthe mobile device to navigate through the saved image information,wherein the user interface element includes at least the representationof the image from the sequence of images and information relating to theimage from the sequence of images.

In accordance with some implementations of the disclosed subject matter,a system for obtaining information relating to presented content isprovided, the system comprising: means for determining that a mobiledevice is associated with a media presentation device over acommunications network, wherein the media presentation device is in anoperating mode that presents a sequence of images; means for receiving auser command that indicates interest in an image from the sequence ofimages being presented on the media presentation device; means fordetermining metadata associated with the image from the sequence ofimages being presented on the media presentation device; means forcausing the metadata associated with the image to be stored with arepresentation of the image from the sequence of images and a useraccount identifier corresponding to a user of the mobile device; meansfor receiving a request to review saved image information correspondingto one or more images in which the user command indicative of interesthas been received; and means for causing a user interface element to bepresented that allows a user of the mobile device to navigate throughthe saved image information in response to receiving the request,wherein the user interface element includes at least the representationof the image from the sequence of images and information relating to theimage from the sequence of images.

In some implementations, the system further comprises means fordetermining the operating mode of the media presentation device, whereinthe sequence of images is caused to be presented on the mediapresentation device in response to determining that the operating modeof the media presentation device is a background mode and wherein mediacontent is caused to be presented on the media presentation device inresponse to determining that the operating mode of the mediapresentation device is a playback mode.

In some implementations, the system further comprises means forreceiving an indication that the mobile device is associated with themedia presentation device over a wireless communications network.

In some implementations, the means for receiving the user command thatindicates interest in the image from the sequence of images furthercomprises: means for receiving audio data that includes ambient soundsin an environment in which the sequence of images is being presented;and means for identifying a voice input that includes a particularsequence of words from the audio data that corresponds to a request toobtain information associated with the image being presented on themedia presentation device.

In some implementations, the means for determining the metadataassociated with the image from the sequence of images being presented onthe media presentation device further comprises: means for obtainingtiming information associated with the received user command; means fortransmitting, to the media presentation device, a request for imageinformation associated with the image being presented on the mediapresentation device based on the timing information; means forreceiving, from the media presentation device, the image information;and means for transmitting a search query for supplemental content,wherein the search query includes the image information.

In some implementations, the system further comprises means for causinga confirmation interface to be presented that confirms the interest inthe image from the sequence of images being presented on the mediapresentation device, wherein the confirmation interface causes therequest for image information to be transmitted to the mediapresentation device.

In some implementations, the system further comprises means forselecting a subset of metadata from the metadata associated with theimage, wherein the subset of metadata and an image identifier thatcorresponds to the representation of the image are stored in a tableassociated with the user account identifier.

In some implementations, the means for receiving the request to reviewsaved image information further comprises means for determining that abrowser interface authenticated with credentials corresponding to theuser account identifier has been accessed, wherein the user interfaceelements is caused to be presented in the browser interface.

In some implementations, the system further comprises: means forreceiving a user selection of the user interface element that ispresenting the representation of the image from the sequence of imagesand the information relating to the image from the sequence of images;and means for causing the browser interface to access a source of theinformation relating to the image.

BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify like elements.

FIG. 1 shows an example of a generalized schematic diagram of a systemon which the mechanisms for storing information associated with contentpresented on a media presentation device as described herein can beimplemented in accordance with some implementations.

FIG. 2 shows an example of hardware that can be used to implement one ormore user devices, presentation devices, and servers depicted in FIG. 1in accordance with some implementations of the disclosed subject matter.

FIG. 3 shows an example of a process for storing information associatedwith content presented on a media presentation device in accordance withsome implementations of the disclosed subject matter.

FIG. 4 shows an example of a display device presenting content and auser device presenting a confirmation interface in accordance with someimplementations of the disclosed subject matter.

FIG. 5 shows an example of a display device presenting content and auser device presenting a user interface for navigating through savedimage-related information in accordance with some implementations of thedisclosed subject matter.

FIG. 6 shows an example of a data flow that can be used in processes ofFIG. 3 among a presentation device, a user device, and one or moreservers in accordance with some implementations of the disclosed subjectmatter.

DETAILED DESCRIPTION

In accordance with some implementations, as described in more detailbelow, mechanisms, which can include methods, systems, and/or computerreadable media, for storing information associated with contentpresented on a media presentation device are provided.

In some implementations, a media presentation device, such as a digitalmedia receiver or media streaming device, can request content to bepresented when the media presentation device is on and outputting videodata but lacks image and/or video content to be presented. For example,when the media presentation device first starts (e.g., before anycontent is requested for presentation), after a predetermined period oftime has elapsed with no activity, etc., the media presentation devicecan request content to be presented. In such an example, content can be,for example, a sequence or a slideshow of images. In a more particularexample, in a background operating mode, the media presentation devicecan execute an application that retrieves and presents images or othermedia content until a presentation request is received (e.g., present aparticular media content item, launch a media content provision service,etc.).

Upon being interested in an image being presented on the mediapresentation device, the mechanisms can detect a user command to saveimage-related information associated with the presented image. Inresponse to detecting the corresponding user command, the mechanisms canretrieve at least image identifying information from the mediapresentation device based on timing information associated with the usercommand. For example, the media presentation device can respond to arequest for image information with an image identifier and imagemetadata associated with an image presented on the media presentationdevice at the time the user command was detected.

The mechanisms can store the image-related information along with useridentifying information associated with a user device in an imagedatabase. This can, for example, allow the user to review and/ornavigate through image-related information via a generated userinterface at a later time, where a user-selected portion of the userinterface can provide the user with an opportunity to review additionalor supplemental content obtained using the image-related information.This can also, for example, allow the user to share saved image-relatedinformation corresponding to images of interest that were presented onthe media presentation device with other users (e.g., users that areconnected in a social sharing service with the user of the user device).

It should be noted that, although the implementations described hereingenerally relate to storing and presenting image information associatedwith an image of interest from a sequence of images presented on a mediapresentation device, this is merely illustrative. Any suitable contentidentifier can be obtained in response to receiving a suitable usercommand and metadata and other information pertaining to the contentassociated with the content identifier can be stored for later review.For example, in response to receiving a suitable user command during thepresentation of video content being presented on a media presentationdevice (e.g., while the media presentation device is detected as beingin an active mode), metadata and other information pertaining to thevideo content being presented on the media presentation device can bestored for later review along with a representation of the video content(e.g., an authorized cover image corresponding to the video content). Ina more particular example, the timing information associated with theuser command can be used to transmit a request to the content provideror any other suitable source to obtain metadata and other informationassociated with the portion of video content presented at the time ofthe user command.

These and other features are further described in connection with FIGS.1-14.

FIG. 1 shows an example 100 of a generalized schematic diagram of asystem on which the mechanisms for storing information associated withcontent presented on a media presentation device as described herein canbe implemented in accordance with some implementations. As illustrated,system 100 can include one or more user devices 102. User devices 102can be local to each other or remote from each other. User devices 102can be connected by one or more communications links 104 to acommunication network 106 that can be linked to a server 120 via acommunications link 112.

Although three user devices 102 are shown in FIG. 1 to avoidover-complicating the drawing, any suitable number of these devices, andany suitable types of these devices, can be used in someimplementations.

System 100 can include one or more presentation devices 110.Presentation devices 110 can be local to each other or remote from eachother. Presentation devices 110 can be connected by one or morecommunications links 108 to communication network 106 that can be linkedto server 120 via communications link 112 and/or user devices 102 viacommunications link 104.

System 100 can include one or more servers 120. Server 120 can be anysuitable server or servers for providing access to the mechanismsdescribed herein for storing information associated with contentpresented on a media presentation device, such as a processor, acomputer, a data processing device, or any suitable combination of suchdevices. For example, the mechanisms for storing information associatedwith content presented on a media presentation device can be distributedinto multiple backend components and multiple frontend components and/oruser interfaces. In a more particular example, backend components, suchas mechanisms for receiving requests to associate user accounts with apresentation device, receiving metadata relating to content beingpresented on a presentation device, requests to present saved imageinformation corresponding to content that was presented on apresentation device, etc., can be performed on one or more servers 120.In another particular example, frontend components, such as mechanismsfor presenting content, presenting confirmation interfaces for savingimage information corresponding to content that was presented on apresentation device, requesting content to be presented, identifyinguser devices that are present, setting user preferences, causing a userdevice to be associated with a presentation device, causing an action tobe performed on a user device and/or presenting the results of such anaction, etc., can be performed on one or more user devices 102 and/orpresentation device 110.

In some implementations, each of user devices 102, presentation device110 and server 120 can be any of a general purpose device such as acomputer or a special purpose device such as a client, a server, etc.Any of these general or special purpose devices can include any suitablecomponents such as a hardware processor (which can be a microprocessor,digital signal processor, a controller, etc.), memory, communicationinterfaces, display controllers, input devices, etc. For example, userdevice 102 can be implemented as a smartphone, a tablet computer, awearable computer, a laptop computer, a portable game console, any othersuitable computing device, or any suitable combination thereof. Asanother example, presentation device 110 can be implemented as a digitalmedia receiver, a media streaming device, a game console, a set-top box,a television, a projector, any other suitable computing device, or anysuitable combination thereof.

Communications network 106 can be any suitable computer network orcombination of such networks including the Internet, an intranet, awide-area network (WAN), a local-area network (LAN), a wireless network,a Wi-Fi network, a digital subscriber line (DSL) network, a frame relaynetwork, an asynchronous transfer mode (ATM) network, a virtual privatenetwork (VPN), an intranet, one or more peer-to-peer connections, etc.Each of communications links 104, 108, and 112 can be any communicationslinks suitable for communicating data among user devices 102,presentation device 110 and server 120, such as network links, dial-uplinks, wireless links, hard-wired links, any other suitablecommunications links, or any suitable combination of such links. Notethat, in some implementations, multiple servers 120 can be used toprovide access to different mechanisms associated with the mechanismsdescribed herein for associating multiple users with a mediapresentation device. For example, system 100 can include a userpreferences server 120 that stores user preferences associated with oneor more users and/or one or more user devices 102, an image informationdatabase server 120 that maintains one or more databases ofcorrespondence between users and/or user devices 102 with which aparticular presentation device 110 is associated (e.g., a database entrythat includes a user identifier corresponding to a user accountassociated with user device 102, an image identifier corresponding to animage in a sequence of images being presented on presentation device110, and metadata or any other suitable image information correspondingto the image being presented on presentation device 110), a contentdelivery server 120 that determines which content to cause to bepresented by the particular presentation device 110, a content deliveryserver 120 that provides metadata or other image informationcorresponding to media content being displayed by presentation device110, and/or any other suitable servers for performing any suitablefunctions of the mechanisms described herein.

In some implementations, user device 102 can be associated with useridentifying information 130. User identifying information 130 canidentify a user of user device 102 and/or can identify user device 102.For example, in some implementations, user identifying information 130can be a token or other data associated with a user of user device 102.For example, the token or other data can identify a user associated witha particular user account of a product and/or service. In a moreparticular example, such a token or other information can include astring of characters that is associated with a particular email addressthat was used to sign in to an application on the user device. Asanother example, user identifying information 130 can be identifyinginformation of user device 102, such as a MAC address, a device ID, aserial number, and/or any other suitable identifying information of userdevice 102. As yet another example, user identifying information 130 canbe a combination of identifying information of a user and identifyinginformation of user device 102.

In some implementations, presentation device 110 can be associated withpresentation device identifying information 140. Presentation deviceidentifying information 140 can identify a user of presentation device110 and/or presentation device 110. For example, in someimplementations, device identifying information 140 can be a token orother data associated with a user of presentation device 110. Forexample, the token or other data can identify a user associated with aparticular user account of a product and/or service. In a moreparticular example, such a token or other information can include astring of characters (which can be, for example, randomly assigned) thatis associated with a particular email address that was used as acredential to log in to an application executing on the presentationdevice. As another example, presentation device identifying information140 can be identifying information of presentation device 110, such as aMAC address, a device ID, a serial number, and/or any other suitableidentifying information of presentation device 110. As yet anotherexample, presentation device identifying information 140 can be acombination of identifying information of a user and identifyinginformation of presentation device 110. In some implementations,presentation device identifying information 140 can include semanticallymeaningful identifying information, such as a user assigned name (e.g.,“UserA's Living Room Streaming Device”).

In some implementations, presentation device identifying information 140can include a persistent identifier for presentation device 110 that canbe assigned based on any suitable conditions. For example, a deviceidentifier of presentation device 110 can be assigned when presentationdevice 110 is initialized and/or reinitialized. In a more particularexample, during initialization, presentation device 110 can contact aserver to request a persistent device identifier. In someimplementations, this device identifier can be assigned by the serversuch that each presentation device has a unique device identifier.Additionally, presentation device 110 can receive a different deviceidentifier upon presentation device 110 being reset or otherwisereinitialized. In some implementations, such a device identifier can beused to associate user identifying information and/or any other suitableinformation (e.g., at a server) with presentation device 110 for lateruse in determining content to be presented using presentation device110.

In some implementations, server 120 can store an image informationdatabase 150. Image information database 150 can include metadata orother image information associated with images and/or other mediacontent being presented on presentation device 110, where such metadataor other image information can be associated with a user of a particularuser device 102 (e.g., associated with user identifying information 130of that user device 102 in response to receiving a user indication ofinterest in an image or other media content being displayed bypresentation device 110). For example, in response to receiving aparticular user command (e.g., detecting the voice command of “save thatscreen”), metadata and other image information corresponding to themedia content being presented on presentation device 110 can be obtainedand stored along with user identifying information of the user device102 or user identifying information of presentation device 110 in imageinformation database 150. In some implementations, metadata and/or otherimage information obtained from presentation device 110 can be used totransmit one or more requests to retrieve additional or supplementalcontent related to the image of interest. Such additional orsupplemental content (or links to additional or supplemental content)can be stored along with the metadata as part of a record in imageinformation database 150. Additionally or alternatively, in someimplementations, image information database 150 can include informationidentifying which users are associated with which presentation devices(e.g., by relating user identifying information 130 and presentationdevice identifying information 140). In some implementations,information in image information database 150 can be organized using anysuitable technique or combination of techniques. For example imageinformation database 150 can be organized as a relational database.

In situations in which the mechanisms described herein collect personalinformation about users, or can make use of personal information, theusers can be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about cached devicedetails on a user's user device, devices discovered on networks to whichthe user device is connected, an address from which a database query issent, a social network, social actions or activities, profession, auser's preferences, or a user's current location), or to control whetherand/or how to receive content from the server that can be more relevantto the user. In addition, certain data can be treated in one or moreways before it is stored or used, so that personally identifiableinformation is removed. For example, a user's identity can be treated sothat no personally identifiable information can be determined for theuser, or a user's geographic location can be generalized where locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular location of a user cannot be determined. Thus, theuser can have control over how information is collected about the userand used by a content server.

In some implementations, information stored in image informationdatabase can be stored such that personal information of a user isobscured. For example, user identifying information 130 and/orpresentation device identifying information 140 can be an assignedidentification number and/or code name and user preferences can beassociated with such an identification number and/or code name.

FIG. 2 shows an example 200 of hardware that can be used to implementone or more of user devices 102, presentation devices 110 and servers120 depicted in FIG. 1 in accordance with some implementations of thedisclosed subject matter. Referring to FIG. 2, user device 102 caninclude a hardware processor 202, a display/input device 204, memory 206and a transmitter/receiver 208, which can be interconnected. In someimplementations, memory 206 can include a storage device (such as acomputer-readable medium) for storing a user device program forcontrolling hardware processor 202.

Hardware processor 202 can use the user device program to execute and/orinteract with the mechanisms described herein for performingpersonalized actions on mobile devices associated with a mediapresentation device that are based on presented media content,associating multiple devices with a media presentation device,controlling presentation of the content on the media presentationdevice, disassociating user preferences from the media presentationdevice, setting user preferences, causing one or more determined actionsto be performed on a user device, etc. In some implementations, the userdevice program can cause hardware processor 202 to, for example,interact with a device executing at least a portion of process 300 asdescribed below in connection with FIG. 3. In some implementations,hardware processor 202 can send and receive data through communicationslink 104 or any other communication links using, for example, atransmitter, a receiver, a transmitter/receiver, a transceiver, or anyother suitable communication device, such as transmitter/receiver 208.Display/input device 204 can include a touchscreen, a flat paneldisplay, a cathode ray tube display, a projector, a speaker or speakers,and/or any other suitable display and/or presentation devices, and/orcan include a computer keyboard, a computer mouse, one or more physicalbuttons, a microphone, a touchpad, a voice recognition circuit, a touchinterface of a touchscreen, a camera, a motion sensor such as an opticalmotion sensor and/or an accelerometer, a temperature sensor, a nearfield communication sensor, a biometric data sensor, and/or any othersuitable input device. Transmitter/receiver 208 can include any suitabletransmitter and/or receiver for transmitting and/or receiving, amongother things, instructions for presenting content, instructions forsetting user preferences, instructions for associating user preferenceswith a presentation device, etc., and can include any suitable hardware,firmware and/or software for interfacing with one or more communicationnetworks, such as network 106 shown in FIG. 1. For example,transmitter/receiver 208 can include network interface card circuitry,wireless communication circuitry, and/or any other suitable type ofcommunication network circuitry, one or more antennas, and/or any othersuitable hardware, firmware and/or software for transmitting and/orreceiving signals.

Presentation device 110 can include a hardware processor 212, adisplay/input device 214, memory 216 and a transmitter/receiver 218,which can be interconnected. In some implementations, memory 216 caninclude a storage device (such as a computer-readable medium) forstoring a presentation device program for controlling hardware processor212.

Hardware processor 212 can use the presentation device program toexecute and/or interact with the mechanisms described herein forperforming personalized actions on mobile devices associated with amedia presentation device that are based on presented media content,associating multiple users with a media presentation device, requestingcontent to present based on user preferences of associated users,request and/or transmit presentation device identifying information 140,etc. In some implementations, the presentation device program can causehardware processor 212 to, for example, interact with a device executingat least a portion of processes 300 and 600 as described below inconnection with FIGS. 3 and 6, respectively. In some implementations,hardware processor 212 can send and receive data through communicationslink 108 or any other communication links using, for example, atransmitter, a receiver, a transmitter/receiver, a transceiver, or anyother suitable communication device, such as transmitter/receiver 218.Display/input device 214 can include a touchscreen, a flat paneldisplay, a cathode ray tube display, a projector, a speaker or speakers,and/or any other suitable display and/or presentation devices, and/orcan include a computer keyboard, a computer mouse, one or more physicalbuttons, a microphone, a touchpad, a voice recognition circuit, a touchinterface of a touchscreen, a camera, a motion sensor such as an opticalmotion sensor and/or an accelerometer, a temperature sensor, a nearfield communication sensor, a biometric data sensor, and/or any othersuitable input device. In some implementations, display/input device 214of presentation device 110 can be omitted. Transmitter/receiver 218 caninclude any suitable transmitter and/or receiver for transmitting and/orreceiving, among other things, requests for content to be presented,content to be presented, signals to determine whether one or more userdevices 102 are present, etc., and can include any suitable hardware,firmware and/or software for interfacing with one or more communicationnetworks, such as network 106 shown in FIG. 2. For example,transmitter/receiver 218 can include network interface card circuitry,wireless communication circuitry, USB input and/or output circuitry,HDMI input and/or output circuitry, and/or any other suitable type ofcommunication network circuitry, one or more antennas, and/or any othersuitable hardware, firmware and/or software for transmitting and/orreceiving signals.

Server 120 can include a hardware processor 222, a display/input device224, memory 226 and a transmitter/receiver 228, which can beinterconnected. In some implementations, memory 228 can include astorage device for storing data received through communications link 112or through other links. The storage device can further include a serverprogram for controlling hardware processor 222. In some implementations,memory 228 can include information stored as a result of user activityand/or activity by a presentation device (e.g., user preferences, useridentifying information 130, presentation device identifying information140, image information database 150, content to be presented, requestsfor content to be presented, user credentials for use in accessingcontent to be presented, etc.). In some implementations, the serverprogram can cause hardware processor 222 to, for example, execute atleast a portion of process 300 and 600 as described below in connectionwith FIGS. 3 and 6, respectively.

Hardware processor 222 can use the server program to communicate withuser devices 102 and/or presentation device 110 as well as provideaccess to and/or copies of the mechanisms described herein. It shouldalso be noted that data received through communications link 112 or anyother communications links can be received from any suitable source. Insome implementations, hardware processor 222 can send and receive datathrough communications link 112 or any other communications links using,for example, a transmitter, a receiver, a transmitter/receiver, atransceiver, or any other suitable communication device, such astransmitter/receiver 228. In some implementations, hardware processor222 can receive commands and/or values transmitted by one or more userdevices 102, presentation device 110, one or more other servers 120,and/or one or more users of server 120, such as a user that makeschanges to adjust settings associated with the mechanisms describedherein for associating multiple users with a media presentation device.Display 224 can include a touchscreen, a flat panel display, a cathoderay tube display, a projector, a speaker or speakers, and/or any othersuitable display and/or presentation devices, and/or can include acomputer keyboard, a computer mouse, one or more physical buttons, amicrophone, a touchpad, a voice recognition circuit, a touch interfaceof a touchscreen, a camera, a motion sensor such as an optical motionsensor and/or an accelerometer, a temperature sensor, a near fieldcommunication sensor, a biometric data sensor, and/or any other suitableinput device. Transmitter/receiver 228 can include any suitabletransmitter and/or receiver for transmitting and/or receiving, amongother things, content to be presented, user preferences, useridentifying information 130, presentation device identifying information140, requests for content, etc., and can include any suitable hardware,firmware and/or software for interfacing with one or more communicationnetworks, such as network 106 shown in FIG. 2. For example,transmitter/receiver 228 can include network interface card circuitry,wireless communication circuitry, and/or any other suitable type ofcommunication network circuitry, one or more antennas, and/or any othersuitable hardware, firmware and/or software for transmitting and/orreceiving signals.

In some implementations, server 120 can be implemented in one server orcan be distributed as any suitable number of servers. For example,multiple servers 120 can be implemented in various locations to increasereliability and/or increase the speed at which the server cancommunicate with user devices 102 and/or presentation device 110.Additionally or alternatively, as described above in connection withFIG. 1, multiple servers 120 can be implemented to perform differenttasks associated with the mechanisms described herein.

FIG. 3 shows an example 300 of a process for storing informationassociated with content presented on a media presentation device inaccordance with some implementations of the disclosed subject matter.

It should be noted that, although the implementations described hereingenerally relate to storing and presenting image information associatedwith an image of interest from a sequence of images presented on a mediapresentation device, this is merely illustrative. Any suitable contentidentifier can be obtained in response to receiving a suitable usercommand and metadata and other information pertaining to the contentassociated with the content identifier can be stored for later review.For example, in response to receiving a suitable user command during thepresentation of video content being presented on a media presentationdevice, metadata and other information pertaining to the video contentbeing presented on the media presentation device can be stored for laterreview along with a representation of the video content (e.g., anauthorized cover image corresponding to the video content). In a moreparticular example, during the playback of a particular video contentitem “Movie A,” a suitable user command, such as “save this and who isthis actor,” and, responsive to the user command at the particular timeduring the playback of “Movie A,” metadata and supplemental contentobtaining using the metadata can be stored for the generation andpresentation of recommendation card interfaces on a computing device. Insome implementations, in addition to storing image-related informationor other information associated with presented media content, themetadata and other information pertaining to the video content beingpresented on the media presentation device can also be shared with oneor more users (e.g., users that have a relationship in a social sharingservice with the user of the user device).

As shown in FIG. 3, process 300 can begin, at 310, by detecting that auser device, such as user device 102, is associated with a mediapresentation device, such as presentation device 110, over acommunications network. For example, a user logged into a user accountcan locate a presentation device on a communications network and cause arequest to associate with the presentation device (e.g., connect toUserA's living room presentation device). In a more particular example,user device 102 can cause a request to associate a user of user device102 (e.g., based on user identifying information 130) with a particularpresentation device 110 to be sent to server 120 executing at least aportion of process 300 in response to and/or as part of a requestinitiated by a user to present particular content on presentation device110. Alternatively, presentation device 110 can cause a request toassociate a user of user device 102 with presentation device 110 to besent to server 120 executing at least a portion of process 300 inresponse to presentation device 110 receiving an instruction from such auser device 102 to perform any suitable action. As yet another example,user device 102 can cause a request to associate a user of user device102 (e.g., based on user identifying information 130) with a particularpresentation device 110 to be sent to server 120 executing at least aportion of process 300 in response to any suitable user actioninitiating such an association.

In some implementations, in lieu of or in addition to a user deviceresponding to a signal from a presentation device to detect presence ofuser devices, the user device can transmit a signal to server 120identifying itself as being in the presence of presentation device 110.

In a more particular example, a client application can be loaded on anysuitable user device, such as a smartphone, a tablet computer, awearable computer, etc. Once the client application is loaded, theclient application can initiate presentation device discovery in someimplementations. For example, presentation device discovery can beinitiated on a network to which the user device is connected. In a moreparticular example, the client application can cause the user device tosearch for presentation devices on a network (e.g., a Wi-Fi network)utilizing the Discovery And Launch (DIAL) protocol. In another moreparticular example, a full discovery protocol can be executed thatcauses the computing device to send a User Datagram Protocol (UDP)multicast message on a network to which the user device is connected. Insome implementations, the UDP can include an M-Search message directedto presentation devices, such as digital media renderers and/or digitalmedia servers, digital media players, or any other suitable presentationdevice that outputs, processes, and/or presents media content. In someimplementations, the UDP multicast message can include an address of thedevice sending the message (e.g., the network address of the userdevice), and can include a time period during which replies are to besent. Such a time period can be any suitable time period, such as onesecond, two seconds, etc., and can be set based on any suitable factors.

As another example, presentation device discovery can be initiated todetermine whether presentation devices are in a proximity of the userdevice. In another more particular example, the client application canexecute a BLUETOOTH Service Discovery Protocol (SDP) and/or any othersuitable SDP that allows a device to discover other devices through ashort-range connection.

It should be noted that, prior to initiating presentation devicediscovery or performing any action on the user device, the clientapplication can provide a user of the user device with an opportunity toprovide affirmative consent or authorization to perform actions on theuser device, such as detecting presentation devices connected to theuser device, retrieving media content associated with the user,performing a second screen action, etc. For example, upon loading theclient application on the user device, the client application can promptthe user to provide authorization for retrieving image identifyinginformation associated with the presentation device and storingimage-related information that can include the image identifyinginformation along with user identifying information in a database. In amore particular example, in response to downloading the clientapplication and/or loading the client application on the user device,the user can be prompted with a message that requests (or requires) thatthe user provide consent prior to performing these actions. Additionallyor alternatively, in response to installing the client application, theuser can be prompted with a permission message that requests (orrequires) that the user provide consent prior to performing theseactions.

In some implementations, process 300 can determine an operating mode ofthe media presentation device at 320. For example, it can be determinedthat media presentation device 110 is currently in a backgroundoperating mode in which a sequence or slideshow of images is presentedon media presentation device 110. In this mode, any suitable mediacontent, which can include one or more images, video content, audiocontent, text, graphics, and/or any other suitable content, can bepresented by the media presentation device. In another example, it canbe determined that media presentation device 110 is currently in anactive mode in which media content is presented on media presentationdevice 110 (e.g., a movie, an audio clip, a video, etc.). Upon detectingthat media presentation device 100 is in an active mode, process 300 canobtain particular types of metadata from media presentation device 110(e.g., location metadata on where the current scene is occurring, peoplemetadata on the actors and actresses appearing in the current scene,keyword metadata relating to the media content being presented, etc.) orcan wait for a user query to determine what types of metadata to obtainfrom media presentation device 110. It should be noted that any suitableoperating mode information can be detected—e.g., an operating mode inwhich only a particular type of content is presented.

In some implementations, along with the retrieval and presentation ofmedia content, which can include one or more images, video content,audio content, text, graphics, and/or any other suitable content,metadata relating to each piece of media content can be retrieved. Insome implementations, metadata can include any suitable informationabout the media content. For example, the metadata can include one ormore topics related to the media content. In some implementations, atopic related to the media content can be “arts,” “sports,” “weather,”“personal photos,” “travel,” “stocks,” “news,” “fashion,” and/or anyother suitable topic. As another example, the metadata can include anysuitable information about the subject of the media content, such as adescription of what is depicted in an image. As yet another example, themetadata can include any geographic information related to the mediacontent, such as a name of a location where the media content wascaptured, a name of a landmark that appears in the media content, etc.In a more particular example, metadata about an image of the EiffelTower can include the phrases “Paris,” “France,” “Europe,” “EiffelTower,” etc. As still another example, the metadata can include anysuitable information about one or more sources related to the mediacontent, such as a social media post, a web page, a URI, etc.

In response to detecting that the user device is associated with themedia presentation device over a communications network and in responseto detecting that the media presentation device is in an operating modethat presents a sequence of images, process 300 can determine whether arequest a user command indicating an interest in an image from thesequence of images has been received at 330. For example, in response todetecting that the user device is associated with the media presentationdevice over a communications network and in response to detecting thatthe media presentation device is in an operating mode that presents asequence of images, an audio input device, such as a microphone on theuser device, can be activated, where audio data is analyzed to detect avoice command indicative of a user's interest in an image beingpresented. In a more particular example, the voice command from theaudio data can include a particular sequence of words, such as “savethis image.” In another more particular example, the voice commandindicating the user's interest in an image can be detected after atrigger term that initiates the request, such as “OK device.” It shouldbe noted that, in some implementations, an audio input device this is apart of or connected to the media presentation device can detect thevoice command from the received audio data and transmit an indication ofthe received voice command to the user device that is associated withthe media presentation device.

In some implementations, in response to determining that the mediapresentation device is in a particular operating mode, process 300 candetect whether a particular user command indicating interest in aportion of media content being presented on the media playback devicehas been received. For example, in response to determining that themedia presentation device is in a background or idle mode in which asequence of images is being presented, process 300 can determine whethera user command, such as “save this image,” has been received. In anotherexample, in response to determining that the media presentation deviceis in an active mode in which media content is being presented by themedia presentation device, process 300 can determine whether a usercommand that includes an instruction to save metadata relating to theportion of media content being presented and includes a query relatingto the portion of media content being presented.

In some implementations, multiple commands can be extracted from theuser command. For example, the user command of “save this and who isthat actor” can be received and parsed into at least two commands—e.g.,“save this” as initiating the command to save metadata and other contentassociated with the time that the command was received and “who is thatactor” (or the terms “who” and “actor”) as using the saved metadata toinitiate a search to answer the query made by the user. In a moreparticular example, in response to receiving this user command duringthe presentation of “Movie A,” process 300 can transmit a search queryto one or more search engines including the metadata retrieved from themedia presentation device that corresponds to the time that the usercommand was received by the mobile device. In continuing this example,the search results responsive to the search query or portions thereofcan be stored. For example, the metadata and supplemental content can bestored for later review and an answer to the received query in the usercommand can be presented to the user in the form of a card interface onthe mobile device. In another example, the search results can be used togenerate card interfaces for later review on the mobile device (e.g.,where the user of the mobile device can navigate through variousinterfaces including content saved in response to received usercommands).

In some implementations, in response to detecting that the correspondinguser command has been received (e.g., “OK device, save this image” froman audio input device), process 300 can obtain image identifyinginformation from the media presentation device. For example, process 300can, at 340, cause an application executing on the user device totransmit a request to an application executing on the media presentationdevice to obtain an image identifier associated with an image that waspresented on the media presentation device. Such a request can include,for example, a time stamp that the user command was received. Inresponse, the media presentation device can determine the image or othermedia content that was presented based on the time stamp of the requestand can transmit image identifying information and any otherimage-related information (e.g., the metadata described above) to theuser device for storage. Additionally or alternatively, the mediapresentation device can transmit image identifying information to theuser device, where the application executing on the user device cantransmit a request to a content source for additional informationcorresponding to the image identifying information (e.g., an imageidentifier, a name of the image, an artist associated with the image, adate associated with the image, a location associated with the image,etc.).

In some implementations, process 300 can request image identifyinginformation for multiple images that were presented at or near the timestamp that the user command was received. For example, the user of theuser device can be presented with multiple images and be provided withan opportunity to indicate a confirmed interest in at least one of themultiple images (e.g., by selecting one or more of the presented imagerepresentations).

In some implementations, process 300 can use the time stamp that theuser command was received to allow the user of the mobile device toresume or return to the particular content being presented. For example,if the user command is received during the playback of video content,the metadata and other image-related information presented on a userinterface can include a selectable option to resume or return to thevideo content at a playback time corresponding to the time stamp thatthe user command was received. In continuing this example, the selectionof the selectable option by the user of the mobile device can cause themobile device to transmit an instruction to the media playback device toresume playback of a particular media content item at a particular time(e.g., by requesting the media content item, selecting a particularplayback time, and casting the content to the media playback device). Inanother example, where the user command was received during thepresentation of a sequence of images, the selection of the selectableoption by the user of the mobile device can cause the mobile device totransmit an instruction to the media playback device to display theimage in the sequence of images that was being presented at the timethat the user command was received.

In some implementations, process 300 can cause a confirmation interfaceto be presented on the user device and/or the media presentation device.For example, in response to detecting that the corresponding usercommand has been received (e.g., “OK device, save this image” from anaudio input device), process 300 can cause a confirmation interface thatprompts the user of the user device to confirm whether to initiate thestorage of image-related information (e.g., a representation of thepresented image, metadata associated with the presented image, links tosupplemental content related to the presented image, etc.). Suchimage-related information can then be stored as part of an album in, forexample, a social media service. Additionally or alternatively, aconfirmation interface can also be presented on the media presentationdevice (e.g., a representation of the image can be presented adjacent toa message indicating that it will be saved and associated with thecorresponding user account).

In a more particular example, FIG. 4 shows an illustrative example of adisplay device 410 presenting an image 420 and a mobile device 102presenting a confirmation interface in response to detecting that thecorresponding user command has been received. In some implementations,display device 410 can be operatively coupled to a media presentationdevice and/or a media presentation device can be incorporated intodisplay device 410. For example, a media presentation device, such as adigital media receiver or media streaming device, can request content tobe presented when the media presentation device is on and outputtingvideo data but lacks image and/or video content to be presented. In someimplementations, media content 420 can be default content associatedwith display device 410, such as a collection of images from one sourceor multiple sources (e.g., a locally stored database of images, imagesstored on the presentation device, a server, etc.). Additionally oralternatively, media content 420 can be content that has been selectedfor presentation based on the presence of user device 102 (e.g., userpreferences associated with a user account authenticated on the userdevice). For example, mobile device 102 can be detected to be within aproximity of media presentation device 410 (e.g., within a predeterminedproximity of media presentation device 410, on a local area network towhich media presentation device 410 is connected, etc.). In someimplementations, the collection of images can be displayed as aslideshow of images. For example, each of the images in the slideshowcan be presented one (or many) at a time for a predetermined period oftime (e.g., fifteen seconds, thirty seconds, etc.).

In continuing the above example, upon detecting that the correspondinguser command to save image-related information has been received (e.g.,“OK device, save my screen”), a confirmation interface 430 can bepresented on a display of user device 102. As shown in FIG. 4, a cardinterface 430 that includes a representation 440 of the image ofinterest that was presented on display device 410 and a message 450 thatrequests that the user of user device 102 confirm the interest in savingimage-related information of the image in representation 440 can bepresented on a display of user interface 102. More particularly, inresponse to detecting an interaction with confirmation interface 430(e.g., by selecting message 450), the storage of image-relatedinformation can be cancelled. Otherwise, in response to not detecting aninteraction with confirmation interface 430 within a predeterminedperiod of time (e.g., five seconds), the storage of image-relatedinformation corresponding to the image in representation 440 can beinitiated. Alternatively, in some implementations, a selection ofconfirmation interface 430 can be required to initiate the storage ofimage-related information corresponding to the image in representation440.

In another more particular example, upon detecting multiple commands tosave metadata related to video content being presented and a queryregarding the video content (e.g., “OK device, save my screen and who isthe actress”), a confirmation interface can be presented that includes arepresentation of the video content, a timestamp of when the multiplecommands were received, and a representation of the query (e.g., Did youask—“who is the actress in this scene of Movie A?”). The confirmationinterface can request that the user confirm each of the multiplecommands.

In some embodiments, the confirmation interface can include a selectableoption to return to the image or the position in the media content thatwas being presented when the user command was received. This can, forexample, allow the user to review the image or review the particularscene in the media content item prior to saving metadata or obtainingsupplemental content.

Referring back to FIG. 3, in some implementations, process 300 canretrieve metadata and other image-related information corresponding tothe image that was presented on the media presentation device at 350.For example, upon receiving image identifying information from the mediapresentation device or any other suitable source, process 300 cantransmit a request for supplemental content corresponding to the imagethat was presented on the media presentation device using the imageidentifying information. In a more particular example, the server canretrieve a data blob or any other suitable information about thepresented image and use the data blob to retrieve metadata associatedwith the image.

In some implementations, the metadata can contain any suitableinformation relating to the presented content, such as one or moretopics related to the presented content, information about the type ofinformation contained in the presented content (e.g., an image, a video,a file type, etc.), information about the subject of the presentedcontent (e.g., a description of what is depicted in an image), a sourcewhere the presented content originates (for example, a social mediapost, a web page, a URI, etc.), information about one or more usersrelated to the presented content (e.g., a user that appears in a photo),information about one or more authors of the presented content, etc. Ina more particular example, when the media content presented on the mediapresentation device is a landscape photograph, the metadata can includean image uniform resource locator, a location of where the photographwas taken, and artist information associated with the user that capturedthe photograph. In another more particular example, when the mediacontent presented on the media presentation device is an image from anews article, the metadata can include a uniform resource locator to thenews article, keywords associated with the content contained in the newsarticle, a source that published the news article, and links to contentthat is related to the news article. In yet another more particularexample, when the media content presented on the media presentationdevice is a food image from a social sharing service, the metadata caninclude an image uniform resource locator, a link to a recipe for makingthe food shown in the food image, a link to comments that have beenprovided on the social sharing service in response to the publication ofthe food image, and a link to related recipes.

In some implementations, in response to receiving image identifyinginformation, metadata, or any other suitable image-related informationfrom the media presentation device, process 300 can generate one or moreadditional requests for supplemental content to other content sources.For example, the image identifying information and/or metadata receivedfrom the media presentation device can be used to generate a searchquery that is transmitted to a search engine. The search resultsresponsive to the search query can be reviewed and a portion of thesearch results can be stored along with the image-related information.For example, search results that are determined to have a popularityscore greater than a particular threshold score can be stored with theimage-related information in an image information database. In anotherexample, search results that are determined to be relevant to the userof the user device can be stored with the image-related information inan image information database, where relevance can be based on contentsources typically accessed by the user account associated with the userdevice, content providing services subscribed to by the user accountassociated with the user device, etc.

In some implementations, the search query that is transmitted to one ormore search engines is generated in response to detecting a query in theuser command. The search query can obtain an answer to the querydetected in the user command and the corresponding search results can beobtained.

At 360, process 300 can associate the received image-related informationwith the user account authenticated on the user device. For example,process 300 can store the received image-related information in adatabase indexed by user account, user device, and/or presentationdevice. In a more particular example, an instruction to storeimage-related information from a client application executing on theuser device can cause a representation of the image that was presentedon the media presentation device, image identifying information, andmetadata associated with the image to be stored in association with auser identifier, identifying information related to the mediapresentation device, and/or other suitable information such that, inresponse to receiving a subsequent request for supplemental informationrelated to images of interest that have been presented on the mediapresentation device, a service can retrieve and/or determineimage-related information to present on the user device. In another moreparticular example, in response to receiving a confirmation from theuser of the user device to store image-related information correspondingto an image that was presented on the media presentation device, arequest to store image-related information that includes a useridentifier (e.g., user account credentials), an image identifier, andmetadata associated with the image of interest can be transmitted to aserver.

Noted that in some implementations in which the mechanisms describedherein collect information about a particular user, the user can beprovided with an opportunity to control whether the mechanisms collectinformation about particular users and/or how collected user informationis used by the mechanisms. Examples of information about a user caninclude the user's interests and identifying information of the user(e.g., a user profile, user credentials, device identification, etc.).Additionally, certain information about the user can be stored locally(e.g., not shared), encrypted, and/or treated in one or more ways beforeit is stored to remove personally identifiable information. For example,the mechanisms described herein can store user preferences and/or userinterests for a particular user with an anonymous user identifier (e.g.,a user identifier that is not associated with the user's name, theuser's username and/or password, the user's email address, etc.). Usingthese techniques, the user can have control over what information iscollected about the user and/or how that information is used by themechanisms described herein.

In some implementations, a portion of the metadata can be selected fromthe received metadata that is associated with the presented image. Forexample, based on user interests and previous user selections associatedwith the user account, a portion of the metadata can be selected forstorage, where the portion of metadata is used to generate a userinterface for navigating through stored image information.

It should be noted that selecting metadata or other image-relatedinformation for storage can be based on any suitable criterion. Forexample, user preferences, which may include a user's stated interests,a user's implied interests, media content that the user has consumed,media content and/or products about which the user has commented onand/or that the user has rated, and/or any other suitable informationabout the user, can be used to selected particular metadata for storagein association with a user account. In another example, a user's impliedinterests, which may be based on user actions such as what types ofmedia content the user consumes, what types of products the user buys,the user's actions with relation to the content and/or products (e.g.,whether the user is engaged with the content/product by commentingand/or “liking” the content/product, a rating given to thecontent/product, etc.), can be used to selected particular metadata forstorage in association with a user account. In some implementations, auser can be given an opportunity to determine which information is usedin determining user preferences. For example, in some implementations,user preferences can be manually entered by a user. As another example,in some implementations, a user can select one or more sources ofinformation that may or may not be used in determining user preferences.

In some implementations, in response to storing the image-relatedinformation in a database indexed by user account, a confirmationmessage can be transmitted to the application executing on the userdevice. Upon receiving the confirmation message, the applicationexecuting on the user device can present an interface indicating thatthe user-related information has been stored (e.g., “Image Saved”).Additionally or alternatively, an interface indicating that theuser-related information has been stored can be presented on the mediapresentation device.

At 370, process 300 can present a user interface that allows a user ofthe user device to navigate through stored metadata and/or otherimage-related information associated with one or more images of interestthat have been presented on a media presentation device. For example,the database of image-related information can be accessed and used tocreate an interface where the user can scroll through image-relatedinformation pertaining to images that the user of the user account hasindicated an interest.

In a more particular example, FIG. 5 shows an illustrative example ofdisplay device 410 that had presented image 420 and mobile device 102presenting a user interface 510 for navigating through savedimage-related information. For example, upon detecting that a particularclient application has been launched or otherwise determining that arequest to review saved image-related information has been received, auser interface 510 can be generated based on data associated with theuser account authenticated on mobile device 102 from an image databaseand user interface 510 can be presented on a display of mobile device102. As shown in FIG. 5, user interface 510 can include a representationof the image of interest that was presented on display device 410 (e.g.,Photo A), image-related information or image metadata retrieved from theimage database (e.g., title information, artist information, a link to apage containing the image, a link to a user profile corresponding to theartist or owner, and a selectable list of media presentation devices. Incontinuing this example, user interface 510 can be used to navigatethrough images that have been presented on a particular mediapresentation device in which the corresponding user command wasreceived.

In some implementations, user interface 510 can allow the user to selectfrom multiple media presentation devices that the user of mobile device102 has associated with. In response, mobile device 102 can present auser interface 510 that includes images and other media content that hasbeen presented on that particular media presentation device.Additionally or alternatively, the user of mobile device 102 canindicate that user interface 510 is to include images and other mediacontent information from all media presentation devices.

In some implementations, process 300 can determine one or more actionsto be performed by the user device based on interaction with userinterface 510. For example, if the presented content includes an imageof a painting, a user selection of user interface 510 can retrieve webpages, news articles, and/or any other suitable content related to thepainting, the artist of the painting, etc., by performing a search basedon the metadata related to the painting. In some implementations, usingthe information from the web pages, news articles, and/or any othersuitable content relating to the presented content, the server canobtain a larger set of keywords relating to the presented content todetermine an action that may be of interest to the user of the userdevice.

The server can transmit a response to the user device for presenting theaction associated with the user interface. In some implementations, theresponse can include any suitable data that can be used to present theaction. For example, the response can include a link (e.g., a uniformresource locator (URL)), a barcode (e.g., a quick response (QR) code),and/or any other suitable mechanism directed to a web page includinginformation related to the presented content, etc. As another example,the metadata can include a snippet of web content (e.g., a web page,text, video, etc.) related to the presented content.

In some implementations, the client application can receive theresponse. In some implementations, the client application can cause theaction related to the presented content to be performed by user device102. In some implementations, the action can include presenting contentusing text, images, icons, graphics, videos, animations, audio clips,hypertext, hyperlinks, sounds, and/or any other suitable content.

In some implementations, user interface 510 can include a selectableoption to resume or return to the video content at a playback time orthe image in the sequence of images corresponding to the time stamp thatthe user command was received. For example, the selection of theselectable option by the user of the mobile device can cause the mobiledevice to transmit an instruction to the media playback device to resumeplayback of a particular media content item at a particular time (e.g.,by requesting the media content item, selecting a particular playbacktime, and casting the content to the media playback device). In anotherexample, the selection of the selectable option by the user of themobile device can cause the mobile device to transmit an instruction tothe media playback device to display the image in the sequence of imagesthat was being presented at the time that the user command was received.

FIG. 6 shows a diagram illustrating an example 600 of a data flow thatcan be used in processes of FIG. 3 among a presentation device 110, userdevice 102, and one or more servers 120 in accordance with someimplementations of the disclosed subject matter.

At 602, a media presentation application executing on media presentationdevice 110 can present content on a display associated with mediapresentation device 110. For example, such content can include aslideshow of images in response to determining that media presentationdevice 110 is in a suitable operating mode, such as a background mode.In a more particular example, in response to detecting that a backgroundmode is currently active on media presentation device 110, mediapresentation device 110 can request particular media content from acontent server to be presented on media presentation device 110—e.g., aslideshow of selected photographs, a slideshow of photographs stored inassociation with a user account of media presentation device 110, etc.In another more particular example, in response to detecting that mediapresentation device 110 is operating in an active mode, mediapresentation device 110 can request particular media content from acontent server to be presented on media presentation device 110—e.g.,video content, on-demand video content, audio content, etc.

At 604, a media presentation application executing on user device 102can associate with media presentation device 110. For example, inresponse to launching a media presentation application on user device102, the media presentation application can identify media presentationdevices, such as media presentation device 110. These can include, forexample, media presentation devices that user device 102 has previouslyassociated with and/or media presentation devices that are connected toa same local network as user device 102.

Alternatively, media presentation device 110 can identify user devices,such as user device 102, that are present and receive user deviceidentifying information of present user devices. Media presentationdevice 110 can identify user devices that are present using any suitabletechnique or combination of techniques. For example, as described above,media presentation device 110 can detect user devices that are connectedto a same local network as media presentation device 110. In someimplementations, a user device can be determined to be present based onany suitable criterion or combination of criteria, such as if the userdevice responds to a message sent by media presentation device 110.

At 606, the media presentation application executing on user device 102can capture audio data using an audio input device (e.g., a microphoneon user device 102). The media presentation application can detect thata user command for saving image-related information for an image beingdisplayed on media presentation device 110 has been received. Asdescribed above, the media presentation application executing on userdevice 102 can detect a particular sequence of words to initiate thestorage of image-related information (e.g., “OK device, save thisimage”).

At 608, in response to detecting the receipt of the corresponding usercommand for saving image-related information for an image beingdisplayed on media presentation device 110, the media presentationapplication executing on user device 102 can transmit a request forimage information to media presentation device 110. For example, thetransmitted request can include timing information as to when the usercommand was received.

In response to receiving the request for image information at 610, mediapresentation device 110 can transmit an image identifier or any suitableimage identifying information, metadata, and other image-relatedinformation corresponding to the image that was presented on mediapresentation device 110 at 612. For example, media presentation device110 can transmit a response to user device 102, where the responseincludes an image URL associated with the presented image, keywords thatdescribe the presented image, links to contact in which the presentedimage is discussed, artist information associated with the presentedimage, etc. The media presentation application executing on user device102 can receive the image-related information at 614.

Metadata can contain any suitable information relating to the presentedcontent, such as one or more topics related to the presented content,information about the type of information contained in the presentedcontent (e.g., an image, a video, a file type, etc.), information aboutthe subject of the presented content (e.g., a description of what isdepicted in an image), a source where the presented content originates(for example, a social media post, a web page, a URI, etc.), informationabout one or more users related to the presented content (e.g., a userthat appears in a photo), information about one or more authors of thepresented content, etc. In a more particular example, when the mediacontent presented on the media presentation device is a landscapephotograph, the metadata can include an image uniform resource locator,a location of where the photograph was taken, and artist informationassociated with the user that captured the photograph. In another moreparticular example, when the media content presented on the mediapresentation device is an image from a news article, the metadata caninclude a uniform resource locator to the news article, keywordsassociated with the content contained in the news article, a source thatpublished the news article, and links to content that is related to thenews article. In yet another more particular example, when the mediacontent presented on the media presentation device is a food image froma social sharing service, the metadata can include an image uniformresource locator, a link to a recipe for making the food shown in thefood image, a link to comments that have been provided on the socialsharing service in response to the publication of the food image, and alink to related recipes.

Alternatively, the media presentation application executing on mediapresentation device 110 can capture audio data using an audio inputdevice (e.g., a microphone connected to media presentation device 110).The media presentation application can detect that a user command forsaving image-related information for an image being displayed on mediapresentation device 110 has been received and, in response, transmitimage identifying information, metadata, and/or any other image-relatedinformation to the media presentation application executing on userdevice 110.

In some implementations, at 616, the media presentation applicationexecuting on user device 102 can obtain supplemental image informationbased on the received image-related information. For example, uponreceiving image identifying information and metadata associated with thepresented image from media presentation device 110, a request forsupplemental content corresponding to the image that was presented onthe media presentation device can be transmitted, where the requestincludes the metadata and the image identifying information as searchterms. In response to the request, the search results or supplementalimage information from the one or more search results can be appended tothe image-related information received from media presentation device110.

At 618, the media presentation application executing on user device 102can cause a confirmation to be presented via a confirmation interface at620. For example, a messaging application executing on user device 102or any other suitable application can present a confirmation interface,such as the one shown in FIG. 4. The confirmation interface can, forexample, provide the user of user device 102 with an opportunity tocancel or confirm an interest in an image presented on mediapresentation device that corresponds with the received user command.

At 622, a confirmation response can be received using the messagingapplication executing on user device 102. For example, the confirmationresponse can be an indication to not save the image-related informationand, in response, the image storage process can end. In another example,the confirmation interface can indicate that the image storage processwill continue unless user interaction with the confirmation interface isreceived (e.g., a user selection of the confirmation interface) and, inresponse to not detecting such user interaction within a given period oftime, the confirmation response can include transmitting the image,image identifying information, metadata corresponding to the image,supplemental content corresponding to the image, and/or any othersuitable image-related information in association with user identifyinginformation.

At 624, in response to receiving the storage request, the image,supplemental content corresponding to the image, and/or any othersuitable image-related information can be stored in an image database inassociation with user identifying information. For example, a socialmedia application executing on user device 102 can receive the requestand can create an album that stores a representation of the image andthe image-related information in an image database that is indexed byuser identifying information. It should be noted that any suitableapplication executing on user device 102 can be used to storeimage-related information associated with images of interest identifiedby the user of user device 102.

In some implementations, in response to storing the image-relatedinformation along with user identifying information in the imagedatabase, the social media application on user device 102 can transmit aconfirmation indication that the image and the image-related informationcan be stored in association with user identifying information in theimage database at 626. The confirmation indication can be transmittedback to media presentation application or any other suitable applicationexecuting on user device 102. In response to receiving a confirmationresponse that the image and its associated image-related information hasbeen stored at 628, a confirmation message can be presented on userdevice 102 and/or media presentation device 110 at 630 (e.g., themessage “image saved” along with a representation of the image can bepresented as a notification on user device 102).

At 640, in response to storing the image-related information and/orreceiving a request to review saved image information, the social mediaapplication or any other suitable application executing on user device102 can present a user interface for navigating through saved imageinformation, such as the user interface shown in FIG. 5. For example, inresponse to receiving a request to review image information related toimages of interest that have been presented on the media presentationdevice, a service can retrieve and/or determine image-relatedinformation to present on the user device and generate a suitable userinterface.

Such a user interface can allow the user of user device 102 to scrollthrough multiple representations of images of interest that have beenpresented on one or more media presentation devices. Each imagerepresentation can be presented along with at least a portion of theimage-related information.

In some implementations, the user interface can include an option toshare an image of interest with other users (e.g., users connected tothe user of user device 102 via a social sharing service).

In some implementations, the mechanisms described herein can includesoftware, firmware, hardware, or any suitable combination thereof. Forexample, the mechanisms described herein can encompass a computerprogram written in a programming language recognizable by one or more ofhardware processors 202, 212 and 222 (e.g., a program written in aprogramming language, such as, Java, C, Objective-C, C++, C#,JavaScript, Visual Basic, or any other suitable approaches). As anotherexample, the mechanisms described herein can encompass codecorresponding to one or more Web pages or Web page portions (e.g., viaany suitable encoding, such as Hyper Text Markup Language (“HTML”),Dynamic Hyper Text Markup Language (“DHTML”), Extensible Markup Language(“XML”), JavaServer Pages (“JSP”), Active Server Pages (“ASP”), ColdFusion, or any other suitable approaches).

In situations in which the mechanisms described herein collect personalinformation about users, or can make use of personal information, theusers can be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about cached devicedetails on a user's user device, devices discovered on networks to whichthe user device is connected, an address from which a database query issent, a social network, social actions or activities, profession, auser's preferences, or a user's current location), or to control whetherand/or how to receive content from the server that can be more relevantto the user. In addition, certain data can be treated in one or moreways before it is stored or used, so that personally identifiableinformation is removed. For example, a user's identity can be treated sothat no personally identifiable information can be determined for theuser, or a user's geographic location can be generalized where locationinformation is obtained (such as to a city, ZIP code, or state level),so that a particular location of a user cannot be determined. Thus, theuser can have control over how information is collected about the userand used by a content server.

In some implementations, any suitable computer readable media can beused for storing instructions for performing the functions and/orprocesses described herein. For example, in some implementations,computer readable media can be transitory or non-transitory. Forexample, non-transitory computer readable media can include media suchas magnetic media (such as hard disks, floppy disks, etc.), opticalmedia (such as compact discs, digital video discs, Blu-ray discs, etc.),semiconductor media (such as flash memory, electrically programmableread only memory (EPROM), electrically erasable programmable read onlymemory (EEPROM), etc.), any suitable media that is not fleeting ordevoid of any semblance of permanence during transmission, and/or anysuitable tangible media. As another example, transitory computerreadable media can include signals on networks, in wires, conductors,optical fibers, circuits, any suitable media that is fleeting and devoidof any semblance of permanence during transmission, and/or any suitableintangible media.

It should be understood that the above described steps of the processesof FIGS. 3, 5, and 6 can be executed or performed in any order orsequence not limited to the order and sequence shown and described inthe figures. Also, some of the above steps of the process of FIGS. 3, 5,and 6 can be executed or performed substantially simultaneously whereappropriate or in parallel to reduce latency and processing times.

It should also be noted that, as used herein, the term mechanism canencompass hardware, software, firmware, or any suitable combinationthereof.

Accordingly, methods, systems, and media for storing informationassociated with content presented on a media presentation device areprovided.

Although the invention has been described and illustrated in theforegoing illustrative implementations, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of implementation of the invention canbe made without departing from the spirit and scope of the invention,which is limited only by the claims that follow. Features of thedisclosed implementations can be combined and rearranged in variousways.

What is claimed is:
 1. A method for obtaining information relating topresented content, the method comprising: determining that a mobiledevice is associated with a media presentation device over acommunications network, wherein the media presentation device is in anoperating mode that presents a sequence of images; receiving a usercommand that indicates interest in an image from the sequence of imagesbeing presented on the media presentation device; determining metadataassociated with the image from the sequence of images being presented onthe media presentation device; causing the metadata associated with theimage to be stored with a representation of the image from the sequenceof images and a user account identifier corresponding to a user of themobile device; receiving a request to review saved image informationcorresponding to one or more images in which the user command indicativeof interest has been received; and in response to receiving the request,causing a user interface element to be presented that allows a user ofthe mobile device to navigate through the saved image information,wherein the user interface element includes at least the representationof the image from the sequence of images and information relating to theimage from the sequence of images.
 2. The method of claim 1, furthercomprising determining the operating mode of the media presentationdevice, wherein the sequence of images is caused to be presented on themedia presentation device in response to determining that the operatingmode of the media presentation device is a background mode and whereinmedia content is caused to be presented on the media presentation devicein response to determining that the operating mode of the mediapresentation device is a playback mode.
 3. The method of claim 1,further comprises receiving an indication that the mobile device isassociated with the media presentation device over a wirelesscommunications network.
 4. The method of claim 1, wherein receiving theuser command that indicates interest in the image from the sequence ofimages further comprises: receiving, using an audio input device, audiodata that includes ambient sounds in an environment in which thesequence of images is being presented; and identifying a voice inputthat includes a particular sequence of words from the audio data thatcorresponds to a request to obtain information associated with the imagebeing presented on the media presentation device.
 5. The method of claim1, wherein determining the metadata associated with the image from thesequence of images being presented on the media presentation devicefurther comprises: obtaining timing information associated with thereceived user command; transmitting, to the media presentation device, arequest for image information associated with the image being presentedon the media presentation device based on the timing information;receiving, from the media presentation device, the image information;and transmitting a search query for supplemental content, wherein thesearch query includes the image information.
 6. The method of claim 5,further comprising causing a confirmation interface to be presented thatconfirms the interest in the image from the sequence of images beingpresented on the media presentation device, wherein the confirmationinterface causes the request for image information to be transmitted tothe media presentation device.
 7. The method of claim 1, furthercomprising selecting a subset of metadata from the metadata associatedwith the image, wherein the subset of metadata and an image identifierthat corresponds to the representation of the image are stored in atable associated with the user account identifier.
 8. The method ofclaim 1, wherein receiving the request to review saved image informationfurther comprises determining that a browser interface authenticatedwith credentials corresponding to the user account identifier has beenaccessed, wherein the user interface elements is caused to be presentedin the browser interface.
 9. The method of claim 8, further comprising:receiving a user selection of the user interface element that ispresenting the representation of the image from the sequence of imagesand the information relating to the image from the sequence of images;and causing the browser interface to access a source of the informationrelating to the image.
 10. A system for obtaining information relatingto presented content, the system comprising: a hardware processor thatis configured to: determine that a mobile device is associated with amedia presentation device over a communications network, wherein themedia presentation device is in an operating mode that presents asequence of images; receive a user command that indicates interest in animage from the sequence of images being presented on the mediapresentation device; determine metadata associated with the image fromthe sequence of images being presented on the media presentation device;cause the metadata associated with the image to be stored with arepresentation of the image from the sequence of images and a useraccount identifier corresponding to a user of the mobile device; receivea request to review saved image information corresponding to one or moreimages in which the user command indicative of interest has beenreceived; and in response to receiving the request, cause a userinterface element to be presented that allows a user of the mobiledevice to navigate through the saved image information, wherein the userinterface element includes at least the representation of the image fromthe sequence of images and information relating to the image from thesequence of images.
 11. The system of claim 10, wherein the hardwareprocessor is further configured to determine the operating mode of themedia presentation device, wherein the sequence of images is caused tobe presented on the media presentation device in response to determiningthat the operating mode of the media presentation device is a backgroundmode and wherein media content is caused to be presented on the mediapresentation device in response to determining that the operating modeof the media presentation device is a playback mode.
 12. The system ofclaim 11, wherein the hardware processor is further configured toreceive an indication that the mobile device is associated with themedia presentation device over a wireless communications network. 13.The system of claim 11, wherein the hardware processor is furtherconfigured to: receive, using an audio input device, audio data thatincludes ambient sounds in an environment in which the sequence ofimages is being presented; and identify a voice input that includes aparticular sequence of words from the audio data that corresponds to arequest to obtain information associated with the image being presentedon the media presentation device.
 14. The system of claim 11, whereinthe hardware processor is further configured to: obtain timinginformation associated with the received user command; transmit, to themedia presentation device, a request for image information associatedwith the image being presented on the media presentation device based onthe timing information; receive, from the media presentation device, theimage information; and transmit a search query for supplemental content,wherein the search query includes the image information.
 15. The systemof claim 14, wherein the hardware processor is further configured tocause a confirmation interface to be presented that confirms theinterest in the image from the sequence of images being presented on themedia presentation device, wherein the confirmation interface causes therequest for image information to be transmitted to the mediapresentation device.
 16. The system of claim 11, wherein the hardwareprocessor is further configured to select a subset of metadata from themetadata associated with the image, wherein the subset of metadata andan image identifier that corresponds to the representation of the imageare stored in a table associated with the user account identifier. 17.The system of claim 11, wherein the hardware processor is furtherconfigured to determine that a browser interface authenticated withcredentials corresponding to the user account identifier has beenaccessed, wherein the user interface elements is caused to be presentedin the browser interface.
 18. The system of claim 17, wherein thehardware processor is further configured to: receive a user selection ofthe user interface element that is presenting the representation of theimage from the sequence of images and the information relating to theimage from the sequence of images; and cause the browser interface toaccess a source of the information relating to the image.
 19. Anon-transitory computer-readable medium containing computer executableinstructions that, when executed by a processor, cause the processor toperform a method for obtaining information relating to presentedcontent, the method comprising: determining that a mobile device isassociated with a media presentation device over a communicationsnetwork, wherein the media presentation device is in an operating modethat presents a sequence of images; receiving a user command thatindicates interest in an image from the sequence of images beingpresented on the media presentation device; determining metadataassociated with the image from the sequence of images being presented onthe media presentation device; causing the metadata associated with theimage to be stored with a representation of the image from the sequenceof images and a user account identifier corresponding to a user of themobile device; receiving a request to review saved image informationcorresponding to one or more images in which the user command indicativeof interest has been received; and in response to receiving the request,causing a user interface element to be presented that allows a user ofthe mobile device to navigate through the saved image information,wherein the user interface element includes at least the representationof the image from the sequence of images and information relating to theimage from the sequence of images.